United States Patent m 

Shaw et al. 



[54] METHOD AND APP ARATUS INCLUDING 
SYSTEM ARCHITECTURE FOR 
MULTIMEDIA COMMUNICATION 

[76] Inventors: Venson Shaw, 11 Yellow Brook RdU 
Marlboro, NJ. 07746; Steven M 
Shaw, 111 Reldyes Ave., Leonia, N.J. 
07605 

[21] Appl. No.: 516,603 

p2] Filed: Aug. 18, 1995 

Related US. Appfication Data 

[63] Continuation of Sen No. 356,4*6, Dec. 15, 1994, aban- 
doned. 

[51] Int CL 6 G06T1/00 

[52] U.S. CI. 370/465; 395/806; 348/390; 

348/586; 358/426; 382/232; 364/514 A 

[58] Field of Search 37(V465^68, 

370/230-231; 395/612, 114, 888, 327-328, 
806-807, 200.01-200.21, 800, 501, 502, 
507, 516, 285; 348084-459, 586, 589; 
358/426-433; 382/166, 232-253, 305; 375/240-241; 

364/514 A, 514 R 




US005706290A 
[ii] Patent Number: 5,706,290 
[45] Date of Patent: Jan. 6, li>98 



[56] References Cited 

U.S. PATENT DOCUMENTS 

5,164,938 11/1992 Jnikevichet al. 37<V231 

5,457,780 KV1995 Shaw et al — 395/502 

5,546,324 8/1996 Palmer et al. 364/514 R 

5,600344 2/1997 Shaw et al. ~. 395/800 

5,611,038 3/1997 Shaw ct al. 395/806 

Primary Examiner— Joseph H. Feild 
Attorney, Agent, or Firm— Clifford G. Frayne 

[57] ABSTRACT 

A controller apparatus and method for executing a plurality 
of control functions for communication of multimedia 
articles including voice, audio, text, still image, motion 
video and animated graphics which incorporates a frame 
memory system which is compatible with multiple standard 
or customized coding algorithmic signals such as H.261. 
MPEG, JPEG, EDTV or HDTV or in multiple incompatible 
video coding equipment employing different video coding 
algorithms can now freely communicate with each other, the 
controller apparatus further including a band width manage- 
ment function for automatically scaling the multimedia 
article signal to conform to available band width. 

30 Claims, 26 Drawing Sheets 
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METHOD AND APPARATUS INCLUDING 
SYSTEM ARCHITECTURE FOR 
MULTIMEDIA COMMUNICATION 

This is a continuation of application Ser. No. 08/356,486, 
filed Dec. 15. 1994. now abandoned. 

FIELD OF INVENTION 

The present invention relates to a method and apparatus 
for improving the efficiency of electronic communication 
and in particular, to a method and apparatus which can 
communicate with available electronic desk top equipment, 
such as personal computers, workstations, video cameras, 
television, VCR's, CD players and telephones and receive, 
store, process and send multiple forms of media information, 
such as sound, image, graphics, video and data, both digi- 
tally and algcrithmicalfy based on a plurality of selective 
band widths. 

BACKGROUND OF THE INVENTION 

Technology allows the individual to communicate with 
others not only by the telephone, but also by telefax 
machines, personal computers and workstations utilizing 
modems and telephone lines and data and video information 
can also be stored and disseminated by means of videotapes, 
compact discs and television monitors. 

There are methods and apparatus available which allow 
for large amounts of data to be reduced and transmitted in a 
very short amount of time, such methods and apparatus are 
known as compressing the data. Similarly, there are methods 
and apparatus available for enhancing the image quality of 
visual and graphic data that has been compressed and is now 
being displayed. For example, see U.S. PaL No. 4,772,947 
to Kono; U.S. Pat No. 4,703350 to Hinman; U.S. PaL No. 
4,727 ,589 to Hirose; U.S. Pat No. 4,777,620 to Shimoni; 
U.S. Pat No. 4,772,946 to Hammer, and U.S. Pat. No. 
4398.256 to Nussmier. 

While the aforesaid patents teach various methods and 
apparatus for compressing and decompressing data and 
enhancing the image quality of the data, none of the afore- 
said patents have directed themselves to the concept and 
structure of a method and apparatus which would commu- 
nicate with and share resources among the telephone, per- 
sonal computer or workstation, video screen and VCR to 
allow the individual to select and convey multiple forms of 
media information such as sound, image, graphics, data and 
live video in an efficient and effective architecture which 
would automatically adjust to available band widths and 
which would be capable of communicating in multiple band 
widths. 

OBJECTS OF THE INVENTION 

An object of the present invention is to define an inte- 
grated process architecture which can accommodate 
communications, both transmission and retrieval, of all 
digitally-coded or algorithmic multimedia information. 

Another object of the invention is to provide for a novel 
system architecture which is flexible and allows control of 
the variable communications band widths and allows for 
flexible combinations of digitalfy<oded multiple media 
information having application to teleconferencing or edu- 
cational instruction. 

A still further object of the present invention is to provide 
for a novel process architecture which not only allows for 
digital coding techniques, but also can interface with tradi- 
tional analog storage or transmission techniques. 
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A still further object of the present invention is to provide 
for a novel, process architecture which allows the user to 
control program and select the appropriate media combina- 
tion either before or during the communication session. 

5 SUMMARY OF THE INVENTION 

An apparatus and method for multimedia communica- 
tions including voice, audio, text still image, motion video 
and animated graphics which permits communications 
10 between multimedia transmitters and receivers and which is 
compatible with multiple standard or customized coding 
algorithmic signals such as H.261, MPEG, JPEG, EDTV or 
HDTV whereby multiple incompatible video coding equip- 
ment employing different video coding algorithms can now 
15 communicate with each other and which includes a recon- 
figurable memory for selectively adjusting the internal file 
format and size so as to be compatible with any available 
band width. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

These and other objects of the present invention will 
become apparent particularly when taken in view of the 
following illustrations wherein: 
25 FIG. 1 is a pictorial illustration of the communication 
system; 

FIG. 2 is a schematic diagram illustrating the overall 
system methodology; 
FIG. 3 is a schematic of the controller' s internal operating 
30 mode for illustrating band width management; 

FIG. 4 is a schematic of the internal circuitry of the 
multimedia communications assembly; 
FIG. 5 is a schematic of the network coinmunications 
35 processor and its design relationship to the transmission 
processor; 

FIG. 6 is a schematic illustrating the communication 
between the host processor, system memory, pixel processor, 
frame memory and display processor; 
40 FIG. 7 is a schematic of the video codec and display 
subsystem; 

FIG. 8 is a schematic illustration of the standard C3F and 
QCIF memory format; 
45 FIG. 9 is a schematic illustration of applicant's scalable 
memory array reconfigurable technique: 

FIG. 10 is a schematic illustrating the pixel processor 
flexibility to various video coding algorithms; 

FIG. 11 is a schematic of the motion processor sub- 
30 systems: 

FIG. 12A illustrates a programmable logic device 
employing cellular array logic architecture: 
FIG. 12B illustrates the implementation of cellular logic 
J3 processing: 

FIG. 12C illustrates gating logic; 
FIG. 13 is a schematic of the multimedia assembly. 

DETAILED DESCRIPTION OF TEE DRAWINGS 
60 Referring to FIG. 1, there is shown a pictorial illustration 
depicting the communication devices available presently for 
the home or office. These include a VCR 102, CD player 
103, telephone 104. television 106, personal computer 108 
and fax machine 110. Each of these communication devices 
63 has a distinct function. The telephone can transmit and 
receive audio and data; a fax machine can transmit and 
receive text documents, a television can receive video broad- 
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casts and audio; and a personal computer can be used for 
many data processing applications. It is Applicant's inten- 
tion to disclose an assembly which can physically commu- 
nicate with these electronic devices to permit them to 
function complimentary with each other and to communi- 
cate with other electronic devices regardless of whether the 
other communication devices were digital ox algorithmic 
and to code and decode automatically to the available band 
width. The communication is accomplished by a multimedia 
communications assembly 112, being of size and shape, 
similar to that of a VCR. The aforementioned electronic 
devices would interconnect with the multimedia communi- 
cations assembly 112 to allow the user/operator to control, 
complement and utilize the functions of the electronic 
devices by means of the multimedia communications assem- 
bly 112. 

FIG. 2 illustrates the overall system operation and meth- 
odology for the multimedia communications assembly 112. 
Assembly 112 makes it possible to exchange a multitude of 
different forms of media objects over a wide range of 
communication networks. Prior art has shown methods and 
apparatus to improve compression and decompression tech- 
niques for individual media types and individual band width 
ranges. However* since video coding algorithms are intrin- 
sically incompatible with each other, there is need for an 
assembly 112 to provide a common interface whereby 
incompatible equipment can freely exchange media objects 
through interfacing with assembly 112. 

The schematic methodology illustrated in FIG. 2 com- 
prises the following major system components. They are a 
network communications processor 202; a transmission pro- 
cessor 204; a pixel processor 206; a motion processor 208: 
a transform processor 210; a display processor 212; a 
capture processor 220; a frame memory 214 and a host 
processor 218. 

The design of the system architecture as described in 
detail hereafter is to gain the ability to interface with 
multiple types of media objects, including audio, still image, 
motion video, text and graphics. As illustrated in FIG. X 
graphics input might possibly be in the form of an RGB 
format 224; VGA format 226; XGA format 228; or SVGA 
format 230. Text media objects could be either in the form 
of a Group 3 format 232; Group 4 format 234; or ASCI 
format 236. Motion media objects may conform either to 
H.261 format 238; MPEG format 240; or other specialized 
formats 242. Still background media objects could be con- 
forming either to JPEG format 244 or other specialized 
formats 234. Input audio media objects could be confann- 
nng to CD audio format 246; voice grade audio 248 or FM 
audio format 250. 

Each media object within a category, namely, audio, still 
image, motion video, text and graphics would be imported 
to a multiplexer 252 dedicated to each category in order to 
identify the input signal and then be directed to a dedicated 
overlay 254 for each category of media object The overlay 
254 provides the ability for the assembly, disassembly, 
deletion, addition and modification of a selected group of 
miiirinwHft objects. The input signals, be they audio, still 
image, motion video, text or graphics, are converted into 
computer object-oriented language format for encoding into 
a frame memory 214 as described hereafter. This conversion 
before storing into frame memory 214 in cooperation with 
the major components of the system described hereafter, 
permit the compilation of selected input signals which have 
been stored in the frame memory 214 to be assembled, 
interpreted and translated to other system formats with 
relative ease as a result of the intelligent memory manage- 
ment capability inherent in this design. 
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The system architecture provides for an interface which 
will enable multiple incompatible video coding equipment 
employing different video coding algorithms to communi- 
cate. This is accomplished through a scalable frame memory 
5 architecture reconngurabie technique (SMART) described 
in FIG. 9. 

In simplistic terms to be described in detail hereafter, the 
design of assembly 112 allows host processor 218 to identify 
the types of input articles during the import stage, the host 

to processor will then instruct the reconfiguration circuit 256 
and the scaler circuit 258 to provide the required down- 
sampling ratio. The media article being imported can then 
conform or be reduced to the internal file format during the 
import stage. The reverse is true during the exporting stage 

is when the media article in the internal file can be enlarged 
and made to conform to the appropriate algorithm for the 
exporting stage. As a result of our smaller internal file size, 
the real time performance requirement of our pixel processor 
206, graphics processor 222, transform processor 210 and 

20 motion processor 208 is reduced Further, the speed and size 
of the frame memory 214 is also proportionately reduced. 
This design allows various coding algorithms to be micro- 
coded at pixel processor 206. 
Assembly 112 also optimizes the video coding for specific 

25 compression ratios in order to meet specific band width 
requirements. In order to adjust the band width to meet the 
various communication network requirements, band width 
controller 260 receives the band width requirement from the 
network communication processor 202, the band width 

30 controller 260 will then instruct the host processor 218 to 
develop the app r opr iate compression ratio in order to meet 
the real time performance requirements. Band width con- 
troller 260 will also interface with transmission processor 
204 in order to import and export the media article at the 

33 appropriate band width. Assembly 112 can program the 
network communication processor 202, the transmission 
processor 204 and the display processor 212 to provide the 
various types of communications interfaces. 
The internal operation modes of host processor 218 

40 permit it to adapt to different compression ratio requirements 
and network band width requirements. As an example, the 
following are some popular network band width interfaces: 

1. Communicating over an analog phone line employing 
4J V32 modem, 9,600 bit per second (bps) band width is 

required, a quarter common immediate frame (QCIF) format 
is displayed at 7.5 frames per second (fps). 

2. Communicating over a digital ISDN D channel at 16 
kilo bits per second (kbs). The user has two options, either 

yj two quarter common intermediate frame (QCIF) formats can 
be displayed at 7.5 frames per second or one quarter 
common utermediate frame can be displayed at 15 frames 
per second. 

3. Communicating over an analog phone line whereby 
55 19,200 bit per second band width is required. The user has 

two options, either two QCIF (common intermediate frame) 
formats can be displayed at 7.5 frames per second or one 
QQF (quarter common intermediate frame) can be dis- 
played at 15 frames per second. 
6o 4. Communicating over switched 56 kilo bits per second 
digital network. Quarter common mtermediate frames with 
three quality level options will be updated at 15 frames per 
second. 

5. Commiinicating over a single ISDN B channel over an 
65 ISDN basic rate interface network, four quarter common 
intermediate frames will be concurrently updated at 15 
frames per second. 
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6. Communicating over a dual ISDN B channel in a ISDN blocks (4 Y's and 2 U/V's). Bach block would consist of 8x8 
basic rate interface network, quarter common intermediate pixels and each pixel would consist of an 8 bit value. The 
frames will be transmitted at 30 frames per second. quarter common intermediate format frame (QCTP frame) 

7. Communicating over a 384 kilo bits per second ISDN 3** would consist of three groups of blocks and these would 
H 1 network, common intermediate frame Will be transmit- $ be identical to those of me OFs 302. 

ted at 15 frames per second. In multimedia assembly 112. host processor 218 has eight 

8. Communicating over a 1344 kQo bits per second Tl W different network interface modes 3*6. The first interface 
network, common intermediate frames (OF) will be trans- node 310 is for 9.6 Kbs analog modems. The second 
mined at 30 frames per second. interface mode 312 is for 16 Kbs ISDN D channel. The third 

As a result of the aforesaid plurality of band widths, ills 10 netWQrk ^ terfaC L m< ^ lc _i 14 h *°f /'V* 5 ^ 

necessary for the umltimediT assembly to continuously ?** , ?? dem ?\ The fourth netw<rt interface mode 316 is 

monitor the processor and network band width availability fai J* T"^ 

and to simultaneously determine the amount of compression f^™* ^18 is far 64 IOjsBDN single B channel, 

or decompression that is required with respect to ttedata in Thcsucth DetW0lk mterfacc modc 320 is for dual B channel 

frame memory 314 to be transmitted. DueTo the variable 15 128 Kbs ISDN BRI network. The seventh network interface 

band width or throughput requirement for each transmission mode 322 is for 384 BDN H1 netwo * me c « hth 

network, only dedicated processor approaches have been "f^^f*"* n>ode 324 is f or 1.544 Mbs ISDN PRI or 

shown in the prior art to meet a specific band width netwonc 

performance. For example, three video conferencing tech- Host processor 218 also has programmable frame updat- 

niques are required at the 112 Kbs, 384 Kbs and 1.544 Mbs 20 in 8 ratc capability 326. Frame updating rate 326 provides 

band width range. The niultimedia assembly disclosed host processor 218 with rive options. They can be either 30 

herein, includes different transceiver pairs for each specific framc second (fps); 15 fps; 10 fps; 7.5 fps or 1 fps. 

network type. The system architecture disclosed herein, and The standard frame update rate 326 for each network 

in particular, host processor 218 in conjunction with band interface mode 306 would be 1 fps for first network interface 

width controller 260 unit scaler 258 and reconfiguration 25 mode 310; 1.5 fps for second network interface mode 312; 

unit 256, can continuously adapt to a variety of network and 2 fps for third network interface mode 314; 6.5 fps for fourth 

processor band width changing situations, for example, network interface mode 316; 7.5 fps for fifth interface mode 

noisy local line condition and network traffic congestion. 318; 15 fps for sixth and seventh interface mode 320 and 

This is possible as a result of the scalable memory archi- ^ 322, respectively and 30 fps for eighth interface mode 324. 

tecture which permits the continuous reprogramming of the In FIG. 3, we have established 30 fps of frame update rate 

internal file format of frame memory 214 so that it is suitable 326 as the default update rate for OF format 302 transmis- 

for the specific band width requirement at that moment sion and 7.5 fps as the default update rate for QCIF format 

During the interframe coding mode 278, after the in com- 304 transmission. The compression ratios illustrated in FIG. 

ing media articles are received, the appropriate frame size 35 10 and described hereafter are for this default update rate. 

262 will be adjusted first frame by frame difference 264 will The OF format 302 system throughput requires 4.6 mega 

then be calculated For consecutive frame processing, an bytes per second (MBS). The QOF formal 304 requires 288 

appropriate motion vector 270 can be derived For selective kilo bytes per second Assuming we use 8 kilo bytes per 

frame processing, due to the difficulty to identify a suitable second as the measuring base for real time video transmis- 

motion vector 270, interpolation techniques 266 can be ^ sion over firm network interface mode 318, the OF format 

employed to simulate frame difference signal. Decision 302 system would require a compression ratio of 576:1 

logic 272 is employed to analyze situation and make a final based upon the CCJIIT H.261 compression standard The 

decision. In case of scene changes, system will be reset to QOF format 304 would require a 36:1 compression ratio, 

intraframe coding mode for further processing. A detailed Similarly, with respect to the other network interface modes 

design of the motion processor 208 is further shown in FIG. 45 306, the compression ratios would be as follows: The eighth 

11. network interface mode 324 would require a OF format 302 

Although our invention entitled '"multimedia", we have compression ratio of 24:1 whereas QOF format 304 would 
been mostly focued on "new hardware and software tech- require a 1.5:1 compression ratio; seventh network interface 
niques M for the "motion video". In addition, we have also mode 322 would require a OF format 302 compression ratio 
shown new techniques how to integrate (overlay) motion 50 of 96:1 and a QOF format 304 ratio of 6:1; fourth network 
video with other media article in order to create a complete interface mode 316 would require a OF format 302 corn- 
multimedia presentation. Since there have been plenty of pression ratio of 658:1 and a QOF format 304 ratio of 41:1; 
prior arts showing techniques to handle other media, Le., CD third network interface mode 314 would require a OF 
audio, fax, telephone, computer graphics, or digital camera. format 302 compression ratio of 1 ,920: 1 and a QOF format 
Also because the performance requirement for these media 55 304 ratio of 120:1; the first network interface mode 310 
types are much less demanding. Therefore, the encoding and would require a OF format 302 ratio of 3 ,840: 1 and a QOF 
decoding of other media types in our invention can be easily format 304 ratio of 240:1. 

implemented in general purpose computer hardware and As a standard operation in Applicant's rnultimedia 

software, embedded hardware controller, or special purpose assembly, single QOF format 304 will be employed for the 

digital-signal processors. 60 first through fifth network interface modes 310, 312, 314, 

FIG. 3 is a schematic illustration of the controller's 316 and 318, respectively. Double OOF format will be 

operating modes far band width management based upon the employed for sixth network interface mode 320 and single 

international compression standard CCTTT H.261. Based OF format 302 or quadruple QOF format 304 sequences 

upon this standard each common intermediate format frame will be utilized for the seventh and eighth network interface 

(OF frame) 302 consists of twelve ( 12) groups of blocks and 63 modes 322 and 324. 

each group of blocks would consist of thirty-three (33) The advantages of Applicant's multimedia communica- 

macro-blocks with each macro-block consisting of six (6) tions assembly 112 and its operation and capabilities will be 
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discussed hereafter. FIG. 4 illustrates a schematic view of YUV 471 4:2:2 472. Hie ability of the captoe processor 220 

the multimedia communications assembly 11Z It consists of to decode the aforesaid formats provide for a convenient 

the following major system components. They are a network interface between the nudbmedia c^un^ns^sembty 

«mrmlcations>oc«sor 202ia transmission processor 112 and the television 106. VCR 102 or vmeo camera 465 

204- a pixel processor 206; a motion processor 208; a 5 The OF 302 formulated YUV 471 signals will first 

transform processor 210; a display processor 212; a capture transfer out of the capture processor 220 and store in the 

processor 220; a frame memory 214 and a host processor frame memory 214. The luminance (Y) signal 474 will be 

218. These system components can be implemented either loaded Into the motion processor 208 to perform motion 

using custom integrated circuit devices, a programmable estimation 475. A motion vector 476 will be developed for 

integrated circuit; microprocessor, microcontroller, digital 10 each macro block 477 and store in the associated frame 

signal processor or software, depending upon the specific memory 214 location. The difference between the new and 

system performance requirement old macro blocks will also be coded in discrete cosine 

The system components are interconnected through a transform (DCT) coefficients 478 using the transform pro- 
system host bus 418 and a high speed video bus 422. The cessor 210. Pixel pocessor 2W will perfom a raster to 
system host bus 418 allows the host processor 218 to control 13 zig-zag scan conversion 460 quantization 462 and VLC 
access and communicate with the system components such coding 458 of the DCT coefficients 478 for each macro 
as the network communication processor 202, the transmis- block 477 of luminance 474 and ctaommanw 473. The 
sion processor 204. the pixel processor 206, and the frame transmission processor 204 wfll format the OT 302 frames 
memory 214. The video bus 422 interconnects the frame into the CCTTT H.261 238 format and attach the aj^opmte 
memory 214 with such components as the capture processor ^ header 481 information. As an example, a frame 302 
220, the display processor 212, the transform processor 210. will partition into twelve groups of blocks 482 and each 
the pixel procesW»6 and the motion processor 208 to group of blocks 482 will consist of trnrty-three macro blocks 
perform high speed video signal processing functions. Both 477 and each macro block 477 will be composed of four 
me systemhost bus 418 and the video bus 422 are luminance signals 474. and one U & V signal 473. The 
bidirectional parallel buses. * network communication processor 202 will provide the 

Due to the real time performance requirements for the control interface to me udecommutucattons network 480 or 

high speed video frame processing, two system-wide inter- to a microwave ^nk.«3. 

connections are implemented. The first is the video pipeline On the receiving side, the serial compressed video bit 

424 consisting of a direct interconnection between the stream 484 wfll be received from the network communica- 

capture processor 220, pixel processor 206, morion proccs- 30 Hon processor 202. The bit stream will be converted from 

sor 208. transform processor 210, frame memory 214 and serial to parallel and decode the appropriate header message 

display processor 212. The second system interconnect 342 481 using the transmission processor 204 L The infomiation 

consists of the direct interconnection between the network will then be sent to the frame n^ory 214 through pixel 

communication processor 202, transmission processor 204, processor 206. Pixel processor 206 will then perform a 

host processor 218 and pixel processor 2*6. In order to 35 variable length decoder 458, z^-^-tMasteT scar, conver- 

facilitate these interconnect operations, first in, first out sion 460 and dequantizanon 463. The YUV 471 macro block 

memory devices 428 are inserted where appropriate. 477 of DCT coefficients 478 wffl be sent to frame memory 

434 toTpdate the memory array. The video random access perform YUV 471 to RGB 224 conversion and generate 
m^^is^viSrith two access ports 436 and 437 45 analog signal from the RGB 224 or thence generate an 8ba 
SL^cess to the random accesribk memory array. VGA 226 color image through color mapping 486 The 

Z^ScoMH algor^employ display processor 212 wfll then 
frequent use of the mterframe coding 440 to reduce band face to various displays such as televuion 106. personal 
widths. Namely, only the frame difference signal 442 will be computers 108 or monitor. 

transmitted. Therefore, the twin memory accesses are so For ease of interface, host processor 218also provries ifor 
required to store both the new frame 444 and the old frame a high speed small computer system interface (SCSI) 488 
448 and to facilitate frame differencing operations 450. In with the external host 487 such as a personal computer or 
mis design, the pixel processor 206 serves as the bus master work station. The advantage of the small computer system 
420 fa the video bus 422 by having the video random access interface 488 is that it provides a system independent 
memory (VRAM) controller 434 function positioned within 55 interface between the external [host 487 and the rnuJtonedia 
the pixel processor 206 core. This allows pixel processor communications assembly 112. Since only simplified con- 
206 the ability to control video bus 422 and to access video trol messages 489 are required to pass between the two 
random access memory pixel storage for pixel level opera- hosts, modifications to the .system .* > provid, sftr various 
tions 454. Pixel processor 206 also is equipped with the bit operation formats such as DOS 491. UNIX 4M ^ r M aan ; 
level manipulation functions 456 such as variable length 60 tosh 492 can easily be accwnphsheA Tht ; high 1 speed small 
coder ^decoder (VLC) 458. scan format converter 460 computer system interface 488 will also allow the transmis- 
and quantization converter 462 These permit the pixel sion of video sequences between the two hosts, 
processor to utilize international video coding algorithms for In the case of high speed digital network communication, 
communicating as discussed hereafter. the communication pipeline is employed to facilitate real 

The capture processor 220 can decode various types of 65 time frame formatting 410, protocol cartrcUng 412, trans- 
analojt video input formats and convert them (eg.. NTSC mission and decoding. The host processor 218 is the bus 
464 PAL 466 SCAM 468. or SVHS 469) to CQR 601 470 master 420 for the system bus 418. Consequently, host 



06/30/2004, EAST Version: 1.4.1 



5.706,290 

9 10 

processor 218 will be able to access to the frame memory checking and correction. Corrected bit streams are then 

214 and/or system memory 216. and monitor progress converted from serial to parallel form using serial to parallel 

through a windowing operation 494. The windowing opera- converter 568 and are stored in the first in and first out buffer 

tion 494 essentially allows a portion of the system memory 428 for further processing. The first in, first out buffer 428 

216 to be memory mapped 495 to the frame memory 214 so 5 is designed into four 32K bits section. Each section allows 

that the host processor 218 can use it as a window to view for storage of 32K bits which is the maximum allowance of 

frame memory 214 status and operations in real time. a compressed QF frame. Therefore, 128K bits in the first in, 

FIG. 5 illustrates the network communication processor first out buffer allows double buffering and simultaneous 

212 and its design relationship to transmission processor transmitting and receiving of the incoming and outgoing 

204. Network communication processor 202 is comprised of 10 video information frames. 

an analog front end transceiver 514, digital signal processor i n order to accommodate the various network 

modem 516 and a buffer memory 518. These network environments, the network communications processor is 

communication processor 202 components are intercon- designed to operate in the following specific speeds, 

nected through a private NCP bus 520. The transmission 9,5 Kbps (Kilo bits per second), 19.2 Kbps, 56 Kbps, 64 

processor 204 consists of a frame formatter 522, a protocol l5 Kbps, 128 Kbps, 384 Kbps, 1,544 Mbkps (mega bits 

controller 524 and an error processor 526. The transmission per second) and 2.048 Mbps. HP will offer three 

processor 204 components and the buffer memory 518 are options as the standard modes of operation. In mode 2, 

interconnected through another private X bus 528. The single OF or four QC3F sequences will be offered at 

bit-serial D bus 530 facilitates the network communication 334 Kbps and higher. In mode 3, two QC3F sequences 

processor 202 and transmission processor 204 communica- w £jj ^ offered simultaneously at 128 Kbps. 

tion through digital signal processor modem 516 and frame When line conditions degrade, the analog front end 514 

formatter 522 sub-systems. The private NCP bus 520, D bus ^^11 become aware of the degradation as a result of incoming 

530 and X bus 528 are designed to facilitate effective data frame synchronous signal 570. Analog front end 514 will 

addressing and transfer in between the sub-system blocks. th en notify the digital signal processor modem 516 and host 

Furthermore, the buffer memory 518, digital signal proces- 25 processor 218. Host processor 218 will men switch from a 

sor modem 516 and protocol controller 524 are intercon- standard operation to an exception operation mode. Host 

nected to the host processor 218 through system bus 418. processor 218 has three options to lower the bit rate in order 

The specific requirement of the bus design, which may to accommodate and correct the degradation. Option 1 

include address 510, data 512 and control 502 sections is would be for the host processor 218 to notify the pixel 

dependent upon the data throughput 504, word size 506 and 30 processor 206 and select a coarser quantization level 572. 

bus contention 508 considerations. The network communi- Option 2 would be to drop the frame update rate and increase 

cations processor 202 implements the DTE 536 function the interpolation rate 574. Option 3 would be to drop from 

while the host processor 218, and transmission processor OF to QQF 576. When the error processor 526 detects 

204, perform the DCE 532 function. This allows the proper more than two single bit errors, the error processor 526 will 

pairing of the DCE 532 and DTE 536 interfaced to a local 35 notify the pixel processor 206 and host processor 218. Host 

customer premises equipment 534 so as to perform confer- processor 218 again has two options. Either pixel processor 

ence control 538, store and forward 540 or band width 206 can request for an retransmission or host processor 218 

management 542. can delete the complete macro block 477 and wait until the 

Within the network communication processor 202 sub- next macro block is sent. Meanwhile host processor 218 will 

system, digital signal processor modem 516 is the local host 40 send the old macro block 308 from the frame memory 214 

controller 544. Analog front end 514 consists of an analog and use it to update the display. 

to digital converter (ADC) 546 and a digital to analog FIG. 6 illustrates the interactions between the front end 
converter (DAC) 548. The analog-to-digital converter 546 communication systems and the host processor 218, system 
samples and holds the analog input signal 550 and converts memory 216, pixel processor 206, frame memory 214 and 
it to a digital bit stream. The digital-to-analog converter 548 45 display processor 212. These interactions are performed 
buffers the digital output bit streams and converts them into through system bus 418. The rocorning video sequence 602 
an analog output signal The analog front end is the front end is first received by a front end demodulator 515. Network 
interface to the telephone network 480 from the system. The communicarions processor 202 and transmission processor 
output digital bit stream from the analog-to-digital converter 204 will decode the control message and header information 
546 is then transferred to the buffer memory 518 for tern- so 606. The pixel processor 206 and transform processor 210 
pcrary storage. The digital signal processor modem 516 will will then transform these sequences from frequency domain 
access this information through buffer memory 518 to to pixel ^ny* 1 '" and store same in the frame memory 214. 
perform line coding functions. Inside the digital signal The display processor 212 performs the appropriate inter- 
processor modem 516 is a progranimabie digital signal polation to display the output video sequence at the selected 
processor 552. Digital signal processor 552 is programmable 55 frame rate. Similarly, the outgoing video sequence 603 can 
allowing for easy implementation of line coding 554 and be prepared through coding of the frame difference 442 for 
control 556 functions for many of the analog front end 514 each macro block 477 to convert from pixel to frequency 
functions. domain to transmit out through front end modulators 514. 

Within the transmission processor 204 sub-system, the Once the incoming video sequence 602 is received and 
frame formatter 522 first received the incoming information 60 stored in the buffer memory 518 the control message and 
frame 558 and header message 481 from the digital signal header 606 information will then be stored in a first in, first 
processor modem 516 and identifies the proper receiving out memory 428 for further decoding by the network corn- 
video coding algorithm types 560. Protocol controller 524 implications processor 202 and transmission processor 204. 
then takes over and starts the appropriate protocol decoding A self -contained micro controller 608 could provide the 
562 procedures. Once the control frame 564 and information 63 frame formatting 610. error processing 612 and protocol 
frame 558 header information are fully decoded, the infer- control functions 524. This would provide service at low bit 
matioD frame 558 is sent to the error processor for error rate applications up to 64 Kbs range. For higher speed 
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applications 16 bit or 32 hit high performance embedded before final addition. The double buffered input 707 and 
micro controllers could be employed. output 710 first in, first out memories and the input multi- 
ple. 7 illustrates a block diagram of the design of the plexer 716 are employed to allow the four stage pipeline 
video codec and display subsystem 702 and its interaction required for the discrete cosine transform operation. Addi- 
with the transmission processor 204 and host processor 218. 3 tional speed may be obtained through the use of additional 
The video codec and display subsystem 702 consists of pixel transform pipeline processor 744 arranged in parallel, 
processor 206. transform processor 210. frame memory 214 Referring to FIG. 8, as background to Applicant * s scalable 
and display processor 212. Pixel processor 2*6 is the host memory array reconfigurable technique to be described 
controller for the video codec and display sub-system 7tZ hereafter, an understanding of the CDF format 302 and QOT 
Pixel processor 206 is also the controller for the video bus 10 format 304 is necessary. These formats are designed for the 
422. Pixel processor 206 communicates with the host pro- transportation of video information over a telecommunica- 
cessor 218 through system bus 418 using its internal host tion network. They are com monly applied by international 
interface circuit 704. Pixel processor 206 also interconnects coding algorithms such as CCTTT H.261 238 and MPEG 
to transmission processor 204 through a first in* first out 240 standards. 

memory buffer 706 using its internal serial interface 708. 15 The CDF format 302 consists of 352 pixels for each 

Pixel processor 206 interfaces and controls frame memory horizontal scan line with 288 scan lines on the vertical 

214 through video bus 422 using its internal VRAM con- dimension. The OF format 302 is further partitioned into 

troller circuit 434. Pixel processor 206 interfaces with twelve groups of blocks 482. Each group of block consists 

motion processor 208 through video bus 422 and with of 33 macro blocks 477 and each macro block consists of 4 

display processor 212 through private DP bus using its 20 Y blocks 474, 1 U block 473 and 1 V block 473 and each 

internal display processor decoder 714. The pixel processor block consists of 64 8-bit pixels. 

206 also interfaces with transform processor 210 through The QOF format 304 consists of 176 pixels for each 

first in. first out memory 707 and input multiplexer 716. horizontal scan line with 144 scan lines on the vertical 

Pixel processor 206 is also required to perform time dimension. The QOF format 304 is further partitioned into 

critical pixel domain video coder and decoder functions 718. 25 three groups of blocks 482, each group of block 410 

These include variable length coder and decoder, run level consisting of 33 macro blocks 477 with each macro block 

coder and decoder, quantization and dequantization, zig-zag consisting of 4 Y blocks 474, 1 U block 473 and 1 V block 

to raster or rastar to zig-zag scan conversion. 473. 

Since most video coding algorithms employ frame dif- Each macro block 477 comprises 384 bytes of YUV data 

ferencing 450 techniques to reduce band width, only the 30 since the frame rate for QF format 302 is 30 fps (frame per 

frame difference signals 442 wfll require to be coded and second) and each OF format 302 frame consists of 396 

decoded Frame memory 214 is designed to store old frames macro blocks. The band width required to send uncom- 

714 and new frames 712 at two discrete section. Old frame pressed OF format 149 frames would be 4.6 mega bytes per 

714 being stored as the reference model while the difference second which is the equivalent to a total of 576 channels of 

between the new and old frames are being updated via a 35 64 Kbs B channels. 

differencing signal 442 which will be either coded for Each QOF format 304 has 99 macro blocks 477 and 

transmission or decoded and added back to the old frame frame updates at 7.5 fps. The system throughput requires 

714 for the reconstruction of new frame 309. 288 KBs which is the equivalent of 36 channels of 64 KBs 

As an encoder, pixel processor 206 will retrieve from the based B channels 802. Therefore, an uncompressed QF 

frame memory 214 these frame differencing signals 442 in 40 format 302 frame transmitting at 30 fps requires 24 Tl lease 

macro blocks 477. Transform processor 210 will perform the lines 804 and the QOF format 304 transmitting at 7.5 fps 

DCT (discrete cosine transform) function 716 to translate requires 1.5 Tl lines 804. As such, 75 micro seconds would 

each of the Y, U, and V block from pixel to frequency be required to code an incoming QF format 304. 1.2 

domain. The pixel processor 206 will apply these discrete milliseconds would be required for each macro block at 7.5 

cosine transforms to the decoder or encoder function before 45 fps. 

forwarding the coded bit stream to the transmission proces- The CCTIT H.261 standard 238 requires a switch from 

sor 204 for transmission. inter to intra frame mode after every 132 frames of trans- 

As a decoder, pixel processor 206 will retrieve these mission in order to avoid accumulative error. This means 

frame difference bit streams 442 from the transmission that in a 30 fps transmission, every 4.4 seconds intra QF 

processor 204 first in. first out buffer 706. apply the decoding 50 format 302 frame coding will be engaged and in QCIF 

procedures, and then communicate with the transform pro- format 304, at 7.5 fps, intra frame coding wfll be engaged 

cesser 210 through its input first in, first out buffer 707. every 17.6 seconds. 

Transform processor 210 will perform the inverse DCT FIG. 9 is a schematic illustration of the scalable memory 
(discrete cosine transform) operation 485 to derive the pixel array reconfigurable technique utilized by Applicant's mul- 
domain values for each Y, U and V block 471. These pixel 55 timedia assembly 112 in order to optimize the performance 
values will be stored in the transform processor output first for encoding QF format 302. To achieve 30 fps updates, the 
in, first out 710 until the pixel processor 206 retrieves the old time required to encode a macro block 404 is 75 microsec- 
pixcl block from frame memory 214. The signal differential onds. A single 8x8 DCT operation will consume 6.4 micro- 
will then be forwarded to the pixel processor to update the seconds. Since it takes 6 DCT operations to complete each 
new values of Y, U and V. 60 4YJU:1V block within a macro Hock 477, the time required 
Transform processor 210 also performs matrix transposi- fox a single hardware device to execute DCT transform 
tion 736, two-dimensional filter 738, matrix multiplication coding will take 38.4 microseconds which would mean that 
740 and matrix addition 742. These are required since there would only be 36.6 microseconds left for other time 
whenever motion compensation techniques are applied, the demanding tasks such as motion estimation, variable length 
old frame 714 must be filtered before it can be added to me 65 coding and quantization. 

new frame difference 442. Additionally, the Inverse Although pipeline 902 and parallel processing 904 tech- 
(Discrete Cosine Transform) 485 output must be transposed niques can be applied to improve system performance such 
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as multiple DCT transform pipeline processors 744 can be 
cascaded in parallel as shown in FIG. 7, this solution Is not 
acceptable for the consumer based mass market 

The scalable memory array reconfigurable technique 
reduces the standard OF format 302 to a modified CIF 
format 906 with slightly coarser resolution and yet retain all 
of the integrity of the standard OF format 392 and QCXF 
format 3#4. The scalable memory array has the option to 
choose between the CIF format 302 or QCTF format 304. 

The modified CTF format 906 provides a 288hxl92v 
resolution 90S and the modified QQF format 907 provides 
a 144hx96v resolution 910. This provides close to the 
original CIF and QQF 302 and 304 quality respectively and 
also maintains the 4:1:1 integrity of the YUV signal 471. 
Each CD? format 302 will still retain twelve (12) groups of 
blocks 482 and each QQF format 151 will still maintain 
three (3) groups of block 482. The macro blocks 477 and 
pixel 912 format will remain the same. Hie only difference 
is that each group of block 482 will now consist of 18 macro 
blocks (9hx2V) while the original OF format 302 group of 
blocks consisted of 33 macro blocks (llhx3v). 

This is accomplished during the input and output color 
conversion process in that COR 601 image 916 input which 
consists of 720hx480v resolution can be downsampled (5:2) 
918 to the 288hxl92v Y resolution and further down- 
sampled 5:1 920 to the 144hx96v U, V resolution. At the 
output display, the Y, U, V can perform 2:5 upsampling 922 
for the Y and 1:5 upsampling 924 for the U and V. The 
significance of this modified OF format 908 design is mat 
the internal processing performance requirement is reduced 
by 46% which means we are now allowed to use slower and 
more economical hardware for encoder processing. 
Meanwhile, memory subsystems, such as frame memory 
214 and first-in, first-out memory 428, can employ slower 
memory devices that reduce costs. 

Secondly, scalable memory array 926 permits the further 
scaling down of our modified OF format 908 to meet either 
application requirements or cost production requirements or 
to simply drop from a higher resolution format to a coarser 
resolution format to meet the real time and coding require- 
ment As an example, the OF frame format could be 
implemented at 144hx96v resolution and a QQF frame 
format in 72hx48v resolution. Consequently, the cniltimedia 
assembly 112 can employ the standard OF format 302 or 
QQF format 304 when cost and performance are acceptable. 
In other instances* the scalable memory array 926 would be 
adopted so that the OF and QQF formats would be adapted 
as per the following frame selection examples. 



Mode 


CIF 


QCIF 


tyib 




352hx288v 


176fa x 144y 






288b x 192v 


144h x96v 


Modified 




144hx96v 


72bx48v 






7Zhx48v 


36b x 24v 






36hx24v 


18bxl2v 


Modified 
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601 standard. Namely, the standard MPEG 188 can provide 
144<&x960v resolution. The significance is now we are not 
only able to run each memory section as a parallel process, 
but we are also able to provide compatibility between the 
two standards MPEG 240 and H.261 238. Now, the MPEG 
standard 240 designed originally only to provide high reso- 
lution motion video playback locally can now be used to 
transmit compressed MPEG programs across the network 
employing the widely available H.261 video codec facilities. 
Hie scalable memory array also enables the user to manage 
and provide the remote transmission of MPEG 240 video 
programs employing conference controller 928, store and 
forward 930 and video distribution 932. 

It is therefore possible to either down sample a com- 
pressed MPEG frame 240 into one of the modified OF 
format 908 or simply send multiple compressed MPEG 
subframes by partition. For example, a 1440hx960v MPEG 
frame 240 can downsample 5:1 into a 288hxl92v modified 
OF frame 908 for transmission and decode and upsample at 
1:5 to display it at standard MPEG resolution at the corre- 
sponding output 

As an example, the following frame formats could be 
utilized to interchange between H.261 238 and MPEG 240 
standards. 



25 
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The scalable memory array also allows the partition of 
frame memory 214 into sections of modified frames to allow 
multiple processes to run in each frame section. As an 
example, a frame memory 214 of 352hx288v size can be 
scaled down to either a single 288hxl92v section; 4 144hx 
98v sections; 16 72hx48v sections: 64 36x24v sections or 
any of the mixed combinations, all of the sections being 
processed in parallel. 

The scalable memory array can also provide remote 
MPEG 240 video playback. Standard MPEG provides four 
times the resolution improvement over the existing COR 
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Mode 


MPEG 


Q-MTCG 


TYPE 


1 


1440b x960v 


720hx480v 


Standard MPEG 


2 


1152b x 768v 


576h x 384v 


Modified MPEO 


3 


576hx 384v 


288bx 192v 


Modified MPEO 


4 


352hx288v 


I76h x 144v 


Standard dF/MPEO 


5 


288b x 192v 


144b x96v 


Modified OF/MPEG 


6 


144hx96v 


72bx48v 


Modified OF/MPEG 


7 


72bx48v 


36bx24v 


Modified QF/MPEG 


8 


36hx24v 


18b x 12v 


Modified OF/MPEG 



35 



The scalable memory array formats have significance in 
that due to their compact size, they become useful in 
representing moving objects in the foreground when the 
background information is still. The background informa- 
tion would be pretransmitted during the intra frame coding 
mode 936, while the different moving objects would be 
transmitted during the interframe coding mode 938. 
Depending upon the size of the moving object the appro- 
priate size of the modified format will be employed. At the 
decoder end, the moving objects will be ovcrlayed with the 
still background context to provide motion sequence. 

The scalable memory array Is particularly suitable to 
progressive encoding of images when band width needs to 
be conserved. The scalable memory array will choose the 
coarser modified OF format to transmit the initial frames 
and then utilize a larger modified CIF format to send 
subsequent frames such that the complete image sequence 
will gradually be upgraded to the original OF quality. 

The scalable memory array controller performs as a result 
of the cooperation between pixel processor 206 and host 
processor 218. Pixel processor 2*6 is the local host control- 
ler for the video codec and display subsystem 702 and the 
host processor 218 is the global host controller for the 
overall system. The pixel processor 206 serves as the bus 
master for video bus 422 and host processor 218 serves as 
the bus master for the system bus 418. Both the video bus 
422 and the system bus 418 are system-wide parallel inter- 
connects. Video bus 422 is specifically designed to facilitate 
the high speed video information transfer among subsystem 
components. 

FIG. 10 illustrates the Pixel processor 206 designing to 
meet the flexible performance for various types of popular 
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video coding algorithms such as the MPEG, H.261 or JPEG. 
Meanwhile, pixel processor 206 can also perform other pixel 
domain-based proprietary methods. While most pixel algo- 
rithms arc either inter 936 or intra 938 frame coding, the 
CCTTT and ISO standard algorithms (MPEG, JPEG and 
H.261) are transformed domain coding methods employing 
fast DCT implementation and inter frame differencing tech- 
niques. Additionally, MPEG and R261 also apply motion 
compensation techniques. 

The pixel processor 206 is equipped with a 24 bit address 
line 1002 to permit it to access 16 mega bytes of program 
memory. The program memory can further be partitioned 
into separate segments with each segment designated for a 
specific coding algorithm. Since pixel processor 306 is 
raicro-prograrnmable , it is relatively easy to update the 
changes while MPEG 240, H.261 238 and JPEG 244 stan- 
dards are still evolving. 

The pixel processor 206 is also designed with parallel 
processing in mind. The micro programmable architecture 
allows multiple pixel processors 206 to couple over video 
bus 420 to provide concurrent program execution for an 
extremely high throughput. This will allow each pixel pro- 
cessor 206 to be dedicated to a coder 1008 function or a 
decoder 1010 function. If 6 pixel processors 206 are 
employed this will allow the concurrent execution of an 
entire macro block 477. Similarly, the multiplicity of pixel 
processors depending upon cost and size could permit the 
process of an entire group of block 482 simultaneously. 

The choice of host processor 218 is somewhat critical in 
that it must be able to provide an interface with the external 
host 1006, it must be able to execute the popular DOS 491 
or UNIX program 490 such as word processing or spread 
sheet programs and it must be economical. A suggested 
choice is intel 80286 or 80386 microprocessors. These 
provide a convenient bus interface with the AT bus which 
has sufficient bus band width to be used as the system bus 
418 of the system. The aforesaid micro-processors also 
provide compatibility with a wide variety of DOS 491 based 
software application programs. Additionally, the small com- 
puter system interface 488 is readily available and capable 40 
of providing high speed interface between the internal 
system bus and the external host 1006. 

FIG. 11 is a schematic illustration of motion processor 
208 subsystems. Conforming to one of the H.261 coding 
options, motion processor 208 is designed to identify and 
specify a motion vector 1102 for each macro block 477 
within the existing luminance (Y) frame 474. The motion 
vector 1102 for the color difference for (U, V) frames 473 
can then be derived as either 50% or the truncated integer 
value of the Y frame. The principle is that for each 16h x 16v 
source macro block 1108, the surrounding 48hx48v area 
1106 of updated new frame 712 will be needed to be 
searched and compared. The new macro block 477 having 
the least distortion will be identified as the destination macro 
block 1104 and the distance between the source and desti- 
nation macro block will be defined as the motion vector 
1102. 

The direct implementation of motion processor 208 
requires that for each of the four blocks 1109 residing within 
the old source macro block 1108 of the existing frame, the 
corresponding destination macro block 1104 centered within 
the new frame must be identified. Therefore, every 
corresponding, surrounding 6hx6v area 1106 of blocks in the 
new frame must be searched and compared with .the old 
macro block reference in order to derive the best match with 
least distortion. This approach will require 589, 824 cycles 
of search and compare operations. Provided the search and 
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compare operations can be fully pipelined, an instruction 
cycle time of 0.13 nano seconds is soil required which is too 
time-consuming for the 75 microsecond per macro block 
real time requirement at 30 fps updates. 

In order to meet such real time performance requirements, 
the motion processor 208 must employ parallel processing 
and multi-processing techniques. The multimedia assembly 
112 incorporates a fine grain, tightly coupled, parallel pixel 
processor architecture 1112 which provides faster speed and 
better results. This is accomplished by partitioning existing 
macro block 477 into 4 8x8 blocks 1109. Four parallel 
processing arrays 1116 consisting of 24hx24v processor 
elements arc configured into nine (9) regions. These nine 
regions of macro processor elements 1114 are tightly 
coupled together. Bach region of the existing frame can have 
direct interconnection and simultaneous access to its eight 
(8) nearest neighboring regions from the corresponding new 
frame. Eftrfr region of macro processing elements 1114 is 
designated to perform various types of pixel domain pro- 
cessing functions for the 8x8 block extracted from the old 
source macro block 1108. 

FIG. 11 illustrates a parallel search method for 8x8 blocks 
residing within the old source macro block 1108. Each can 
conduct simultaneous match and compare operations with 
all of their nine nearest neighboring blocks. The outputs of 
the nin e matching operations are first locally stored at the 
corresponding regional pixel processor arrays 1116. They 
are then shifted out and summed at the output accumulator 
1118 and adder circuits 1120. The results are then compared 
using the comparator circuit 1122 to obtain the best match. 
The physical distance between the new macro block which 
results in the best match and the old reference macro block 
will be applied as the motion vector for the old luminance 
macro block. 

The regional pixel processor array 1116 can be reconfig- 
urable and is designed based upon nine banks of processor 
element arrays 1126. Each processor element array 882 
consists of sixty-four processor elements 1128. The nine 
banks of processor element arrays 1126 are interconnected 
through shift registers 1130 and switches 1132. In a three- 
dimensional implementation, a vertically-cascaded proces- 
sor array 1138 crossbar switch array 1134 and shift register 
array 1136 can be implemented. Additional layer such as 
storage array can be added to provide the additional func- 
tionality. This array will be extremely powerful when multi- 
layered packaging becomes available for the chip level 
modules and intergrated circuit technologies. 

A two-dimensioiial pixel processor array 1116 can also be 
designed using nine banks of processor element arrays 1126 
equipped with peripheral switches 1132 and shift registers 
1130. The switches 1132 can be reconfigurable to guide 
direction about the date of flow where the shift registers 
1130 can transfer data from any processor element array 
1126 or input to any other processor element array 1126 or 
output Both switches 1132 and shift registers 1130 are byte 
wide to facilitate parallel data flow. The processor element 
arrays 1126 were designed based upon an 8x8 array of 
simple processor elements 1128. 

The processor element arrays 1126 are designed fox 
interconnection among the processor elements so that recon- 
figuration can be accomplished to meet different application 
needs. The processor elements 1128 are designed so that 
each can be programmed to execute simple instructions. 
Each processor element 1128 consists of a simple ALU 1140 
which can execute simple instructions such as add, subtract 
load, store, compare, etc 

FIG. 12A illustrates the design example of a program- 
mable logic device 1201 which employs a cellular array 
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logic architecture. This figure is used to demonstrate the 
functionality and physical design of the device. The practical 
size for an NxN array is dependent upon the application 
requirements and the state of the art of the implementing 
technology. 

FIG. 12B illustrates the practical implementation of a 
cellular logic processor element 1204 using a charge couple 
device 970 technology. The objective is to provide an 
intergrated image sensor array 1206 with the digital prepro- 
cessing capabilities so that image coding for the macro 
blocks and pixel domain image coding functions can be 
performed. The other objective is to allow the implementa- 
tion of on-chip parallel image sensing and parallel image 
processing 976 utilizing the same or compatible technology. 
The Cellular array logic architecture illustrated in FIG. 12B 
arc useful that they can implement fine grain, tightly- 
coupled parallel processing systems. They employ single- 
instruction-multiple-data 1209 or multiple-instruction- 
multiple-data 1210 techniques to provide system throughput 
where traditional sequential computing fails. 

Many cellular array processors have been designed in the 
past Most of them employ a processor array which consists 
of a matrix of processor elements 1128 and switch arrays 
1134 which can provide programmable interconnect net- 
works among the processor elements. These cellular array 
processors are extremely expensive. 

The design illustrated in FIG. 12B is based upon a much 
simpler architecture, the design being dedicated only to 
image processing and coding applications. The major objec 



tion 1226 can be performed by presetting the threshold value 
1234 and then shifting and quanti sizing the corresponding 
transform domain coefficients based upon the zig-zag scan 
format at each low, medium and high frequency regions. The 
threshold value can be reprogrammed to adjust the quanti- 
zation level. The advantage is that as soon as the input image 
is detected sampled and thresholded, several pixel domain 
preprocessing functions, such as frame differencing and 
motion estimation, can be performed right away. The dif- 
ferencing macro blocks will be sent to transform processor 
210 to perform DCT operation 1224, the output of the DCT 
coefficients can further be reloaded into the processor ele- 
ment array to perform quantization. When band width 
reduction control 260 is required, initial thresholding is 
combined with a coarser quantization 1226 level to reduce 
the image resolution. When the system demands faster 
performance, multiple parallel processor element arrays can 
be cascaded to perform concurrent macro block operations 
such as frame differencing, motion processing and quanti- 
20 zation. 

The advantage of charge couple device technology 1202 
is its suitability for image processing, multiplexing, and 
storage operations. This can be done both in the analog and 
digital domain. Therefore, depending upon the application 
25 requirement, both analog processing 1238, digital process- 
ing 1240 and memory functions using these processor 
element arrays 1126 can be accomplished. 

FIG. 13 is a schematic illustration of the functional model 
architecture in order to simplify the functional processes 
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block pixel domain processing functions or motion process- 
ing. 

FIG. 12A is employed to demonstrate how frame differ- 
encing functions can be performed for each of the incoming 
sub-image macro blocks 477. For illustration, 3x3 array is 
used to represent macro block sub-image 477 which, from 
the current frame, is first shifted into the processor element; 
the corresponding macro block sub-image of the previous 
frame 1218 is then loaded into the processor element and the 



principal functional elements comprise a band width man- 
ager 1300, a formatter 1302, a pixel-domain-codec encoder 
1304 coupled with a pixd-domain-codec decoder 1306, a 
transform-doinain-codec encoder 1308 coupled with a 
35 transfcffm^main-codec decoder 1310, a network-domain- 
codec encoder 1312 coupled with a network-domain-codec 
decoder 1314 and a controller 1316. 

The band width manager 1300 provides band width 
control capability wherein a two-dimensional band width - 
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comparison functions are performed between the two macro 40 over-lay-lookup-table (BOLUT) can be constructed to map 
blocks to detect if there is any frame difference. Provided the the specific band width ranges , i.e., 2.4 Kbs to 100 Mbs, et 



difference is larger than the preset threshold value, the macro 
blocks will be marked and the macro block marker 1242 and 
macro block difference 1244 between the two frames will be 
stored In frame memory 214. If there is no difference, the 45 
current frame macro block value 1216 will be deleted and 
the previous frame macro block value 1218 will be used for 
display updates. 
If an excessive number of macro blocks 477 are identified 



at into selective options of media combinations such as 
overlay in the audio, video, text and graphics with various 
types of quality and resolution. 

Additionally, during noisy communication environments, 
the band width manager 1300 function is to constantly 
monitor the network to detect abrupt network band width 
changes caused by local line degradation or network traffic 
congestion. The band width manager 1300 will respond by 
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with frame difference, then a scene or mumination change 50 adjusting the media combinations to accommodate the avail 



has occurred and macro block processor 1220 will notify 
host processor and pixel processor 206 and switch the 
operation from interframe coding 1227 to mtraframe coding 
1228. The significance is that while incoming images sensed 



from the camera, the specific macro blocks with the frame 55 parallelism for time-sharing. 
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able band width. 

During stable communication environment, band width 
manager 1300 operates to reconfigure the different band 
widths specified by the network providing upgradabflity and 



differencing can be identified and stored. Consequently, in 
the interframe coding modes 1227, only those macro blocks 
477 requiring motion estimation and compensation 1222, 
transform coding 1229 or quantization 1226 will be marked 



The formatter 1302 cornmunicates with the band width 
manager 1300 to ascertain the band width availability for 
incoming or outgoing signals. Hie formatter translates this 
external information into an internally-operating format 
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sequence of the current frame. In the case of scene or 
iUumination changes, enough macro blocks will be detected 
with frame differencing that the system will automatically 
switch to intraframe coding mode 1228. 

FIG. 12B illustrates additional pixel domain processing 65 
functions including low pass filtering 1230; high pass fil- 
tering 1232 and quantization 1226. The variable quantiza- 



reconfigure the internal processer and frame memory struc- 
ture pursuant to the dircctioDS of the formatter. This allows 
the external format to be translated into a suitable internal 
format to provide system compatibility. The scalable- 
mernory-array-reconfigurable-technique (SMART) as dis- 
cussed in FIG. 9 is capable of translating a programmable 
internal format in compliance with a wide variety of intcr- 
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natiooal standard and custom video coding algorithms such 
as MPEG. H.261. JPEG and vector quantization. Formatter 
1302 identifies the transmitting or receiving coding 
algorithms, derives their specific format requirements and if 
these external format requirements are different from the 5 
current internal formats, the formatter reformats the hori- 
zontal and vertical resolution which results in a separate 
internal format which is compatible with the external for- 
mat. These internal format operations, such as the reduction 
of the horizontal and vertical resolution, are performed by lQ 
employing interpolation and down sampling techniques or 
up sampling techniques. The formatter 1302 also communi- 
cates with the frame memory so that the frame memory is 
aware of the internal format to be stared. This allows the 
formatter 1302 in conjunction with the scalable memory 
array configurable technique to formulate a scalable proces- 13 
sor and frame memory architecture so that (he internal 
processor and frame memory can be continually adjusted in 
order to reconfigure or modify a suitable internal format for 
any type of external format either being received or sent by 
the network-domain-codec 1314. 20 

The netwark-domam-codec encoder 1312 and decoder 
1314 are used to provide line coding and decoding functions. 
Network domain codec decoder 1314 would receive net- 
work transmissions via its front end transceiver 1320. It 
would then perform protocol procedures 1322, network 25 
communication procedures 1324, variable length coding 
1326, run length coding 1328 and filtering 1330. The result- 
ant transform coefficients and pixel data will then be for- 
warded to either pixel-domain -codec decoder 1306 or 
transform-domain-codec decoder 1310. The network- 30 
domain-codec encoder 1312 would receive encoded pixel 
data or transform coefficients from the other encoders and 
convert them into serial codes for network transmission 
performing functions similar to the network domain codec 
decoder 1314. Simultaneously, band width manager 1300 35 
will interface with encoder 1312 and decoder 1314 to 
exchange protocol control and applications information 
regarding band width availability. 

The pixel-domain-codec encoder 1304 and decoder 1306 
are designed for custom coding algorithms such as vector 40 
quantization, pixel domain operations for the OCT trans- 
form based standard coding algorithms such as MPEG, et al, 
pixel domain operations for motion compensation and image 
post-processing functions and analysis and preprocessing 
techniques for video coding. Thus, the pixel-domain-codec 45 
provides for pixel domain preprocessing 1332, pixel domain 
coding 1334, image processing 1336, color space conversion 
1338, pixel interpolation 1340, vector quantization 1342 and 
color lookup mapping 1344. 

The transform-doinak-codec encoder 1308 and decoder 50 
1310 are specifically designed for forward and inverse 
transformation operations required by the international stan- 
dard coding algorithms such as MPEG, et al. Transform- 
domain-codec encoder and decoder 1308 and 1310 also 
provide forward and inverse transform-based operations 55 
such as Hart transform and Hadamard transform. 
Additionally, generic matrix operations and post-matrix 
operations, such as scan conversion, quantization and nor- 
malization techniques, are performed by the transform- 
domain-codec. 60 

The controller 1316 comprised of either a single or 
plurality of local host processors which manage the instruc- 
tion sequencing and system control functions for data 
transfer, memory management input/output interfacing and 
processor pipelining. 65 

In FIG. 4, we demonstrated a host processor used to 
manage the communications pipeline, the network domain 



codec and the system memory. It also performed general 
administrative tasks and controlled the system bus and 
access to other subsystem buses while communicating with 
the band width manager 1300. 

A second controller is a single or plurality of pixel 
processors used to manage the video pipeline, the scalable 
memory array reoonfigurable technique, frame memories, 
formatters and display processing. Additionally, the pixel 
processor is used to perform pUel-domain-codec encoding 
and decoding functions and can be used in multiples in order 
to facilitate macro block and group of block processing. 
Similarly, a single or plurality of transform processors can 
be employed as coprocessor for the pixel processors, in 
performing transform-domain-codec encoding and decoding 
functions. 

All network transmissions or receiving functions would 
first pass through the netwoik-domain-codec and then be 
directed to the pixel-domain-codec or transform-domain- 
codec after suitable formatting. The media information 
could then be displayed via the pixel -domain-codec decoder 
1306. Origination signals from either storage, camera, TV or 
CD would be subjected to frame differencing 1364 and 
frame image capture 1366 before being encoded by pixel- 
domain -codec encoder 1304. These origination signals 
could then be transmitted via network-domain-codec, 
encoder 1312 dependent upon the band width manager 1300 
and controller 1360 monitoring of band width availability. 

While the invention has been described with reference to 
its preferred embodiment thereof, it will be appreciated by 
those of ordinary skill in the art that various changes can be 
made in the process and apparatus without departing from 
the basic spirit and scope of the invention. 

I daim: 

1. A controller apparatus for executing a plurality of 
control functions for coimnunication of multimedia articles 
including voice, high quality audio, text, still image, motion 
video and animated graphics between multimedia equip- 
ment and communications networks, selectively providing 
receiving functions, retrieving functions, and transmitting 
functions, wherein said controller is compatible with mul- 
tiple band widths and algorithmic coded signals, said con- 
troller providing a scalable architecture for communication 
of mnhimedia data, formatting from internal format to 
external format in compliance with any selective one of a 
plurality of international standardized and customized coded 
algorithms, and band width determination of available dis- 
play band widths according to frame rate and available 
transmission band widths, said controller comprising: 
means for receipt of an external network signal, said 

means comprising a network domain decoder, 
means for receipt of a local origination signal; and means 
for image processing of said origination signal for 
storage coding simulation, and transmission; 
a first control means and format means for selectively 
formatting the format of said external network signal 
and local origination signal to a compatible first inter- 
nal format in compliance with internal processing and 
memory capabilities; 
means for decoding said external network signal accord- 
ing to said first internal format, said means selectively 
comprising a transform domain decoder alone and in 
combination with a pixel domain decoder, 
a second control means and format means for selectively 
formatting said external network signal and said origi- 
nation signal to a second internal format in compliance 
with a selective plurality of coding algorithms for 
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processing, storage and transmission, said means fur- 
ther comprising an encoding means which comprises a 
pixel domain encoder alone and in combination with a 
transform domain encoder for encoding said external 
network signal and said local origination signal; 
means for reconfiguring the memory means for selec- 
tively associating with said apparatus encoders and 
decoders for directing said first and second format 
means; 

band width management means, in communication with 
said second format means and said control means, for 
directing available run time transmission band widths 
and appropriate compression ratios for selectively for- 
matting said second internal format to external trans- 
mission format in compliance with available transmis- 
sion band width range, said band width management 
means continuously moderating and correcting said 
selective formatting of said second internal format to 
said second external transmission format based on 
transmission band width availability; 

means for transmission of said internally formatted exter- 
nal network signal and local origination signal accord- 
ing to said external transmission format, said means 
comprising a network domain encoder. 

2. A controller apparatus in accordance with claim 1 
wherein said network domain decoder for receipt of said 
external network signal selectively comprises a front and 
transceiver, protocol processor; network communications 
processor, variable length decoding processor; run length 
decoding processor; and filtering processor for sampling 
said external network signal, said network domain decoder 
in communication with said first control means and said 
format means to convert the horizontal and vertical resolu- 
tion of said external network signal to an internally-defined 
file format size for reducing said internal real time video 
processing and frame memory updating requirements. 

3. A controller apparatus in accordance with claim 1 
wherein said second controlling means and said second 
formatting means, in communication with said network 
domain encoder and decoder selectively codes and decodes 
said incoming signal based upon the availability of custom- 
ized and international standardized algorithms and selec- 
tively performs motion compensation processes and motion 
simulation processes to improve quality of said motion 
video article, said processes include pattern match, edge 
detection, image enhancement, color mapping and pixel 
interpolation of said external network and local origination 
signal, 

4. A controller apparatus in accordance with claim 1 
wherein external network signal is selectively transferred to 
said transform domain decoder and said pixel domain 
decoder for decoding; said transform domain decoder selec- 
tively providing denormalization, dequantization, scan 
conversion, matrix operation, inverse Haar transform, 
inverse Hadamard transform and inverse discrete cosine 
transforms on said external network signal and said pixel 
domain decoder selectively performing color mapping, vec- 
tor dequantization, pixel extrapolation, color space 
conversion, image processing, pixel domain decoding, and 
pixel domain preprocessing on said external network signal 
for formatting of said external network signal and conver- 
sion to said first internal format for either storage processing, 
display or transmission. 

5. A controller apparatus in accordance with claim 1 
wherein said means for image preprocessing of said local 
origination signal comprises an integrated circuit subsystem 
further comprising: 
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means for performing integrated parallel sensing and 
storage wherein a plurality of input frames for said 
local origination signal can be sensitized, digitized and 
stored; and 

5 means for performing detection of instantaneous scene 
changes wherein only the corresponding frame will be 
registered as intraframe, wherein all other frames will 
be registered as interframes, said detection means fur- 
ther comprising means for moderating the threshold 

10 level for proper determination of said scene changes; 
and 

means for performing in analog and digital forms parallel 
image preprocessing for said interframes whereby each 
interframe image can be retrieved and compared to 
13 develop a plurality of corresponding frame differencing 
articles whereby only said frame differencing articles 
are required for further processing and encoding 
including motion estimation, discrete cosine transform, 
quantization, and huffman-like variable length coding. 
20 6. A controller apparatus in accordance with claim 5 
wherein said means for image preprocessing for each said 
interframe further comprise a means for motion estimation, 
motion simulation and motion compensation operations on 
motion video articles comprising: 
means to perform edge detection operations for each of 
said frame differencing articles wherein a selective 
plurality of detected edge articles are derived to iden- 
tify said motion video articles during motion estimation 
^ and compensation operations; 

means to perform feature extraction operations for each of 
said frame differencing articles wherein said extracted 
specific features for said frame differencing articles can 
identify from said motion video articles during said 
35 motion estimation process; and 

means to perform forward, backward and random refer- 
ence of selective interframes employing said selective 
features and said selective detected edges for said 
frame differencing articles. 
40 7. A controller apparatus as set forth in claim 5 wherein 
said motion video articles include a plurality of moving 
foreground pixels overlaying can be referenced using trans- 
mission of a single or plurality of preceding intraframes 
wherein said small moving foreground pixels for subsequent 
45 plurality of interframes can be referenced using frame 
differencing techniques wherein each group of said moving 
foreground pixels are characterized employing said frame 
differencing of a plurality of said detected edges and a 
plurality of said extracted features, 
so 8. A controller apparatus in accordance with claim 7 
wherein a motion video article can be applied with one or 
more pattern searching and recognition algorithms to derive 
alternative reference motion vectors wherein said motion 
vectors can forward, backward and random reference said 
55 motion video article during said interframe sequencing said 
motion vector can be represented by the interframe displace- 
ment of symbolic representation further comprising a plu- 
rality of detected edges, frame differences and extracted 
features for each corresponding motion video article, said 
60 symbolic representation and alternate motion vector can be 
applied for transmission simulation, coding, storage and 
processing of said motion video articles. 

9. A controller apparatus in accordance with claim 5, 
wherein said means for interframe image preprocessing 
65 further comprises parallel processor means for performing 
pixel level operations for an nxn pixel block sub-image, said 
means comprising an mxm array of said parallel processor 
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dements wherein n and m are integers and said parallel 
processor means can retrieve, decode and execute instruc- 
tions for pixel input data from internal memory; and inter- 
connection means for selective communication between 
each of said parallel processors whereby said pixel data 
inputs and incoming control messages can be received from 
said selected parallel processors and said pixel data outputs 
and outgoing messages can be broadcast to said selected 
parallel processing neighbors. 

10. A controller apparatus in accordance with claim 5 
wherein said means for interframe image preprocessing 
further comprises a processor array configured as an nxn 
cellular logic array processor where n is typically of value in 
a range of between 8 and 32 wherein said cellular logic array 
processor can be employed to perform a plurality of block 
processing operations for said nxn pixel block of sub-image, 
said processor array further comprising: 
a means for frame differencing between interrrames 
wherein a group of nxn difference blocks are identified, 
registered and stored; 
a means for template matching and article recognition 
operations for each of said nxn pixel blocks of said 
sub-image article extracting from said input image and 
rornparing with selective nxn reference data stored 
previously in memory; 
a means for determination of threshold level for selective 
matching said sections of said sub-image article to 
identify a motion vector article; 
a means for motion estimation operation which can be 
applied to each of said nxn sub-image blocks for 
comparison between said previously-displayed frame 
article and said current frame article, said nxn sub- 
image performing matching functions at said frame 
differencing locations for generating horizontal and 
vertical distance for representing said motion vector, 
a means for edge detection for each of said nxn blocks of 
said sub-image and means for edge enhancement for 
each of said decompressed Image blocks prior to 
display, said edge detection for deriving said motion 
vector; 

a means for feature extraction and comparison of said 

previously-displayed frame with said current frame; 
Hadamard transform means for each of said nxn blocks of 
said sub-image wherein each of said nxn blocks is 
represented by its pixel domain transform coefficient 
multiplied by the nxn Hadamard coefficient matrix 
thereby reducing transmission band width through 
pixel domain band width compression; and 
frequency domain transform means including discrete 
cosine transform means wherein each of an 8x8 frame 
differencing sub-image block can be multiplied by an 
8x8 transform coefficient block to derive 8x8 blocks of 
frame differences identified by said frequency domain 
transform coefficients. 
11. A controller apparatus in accordance with claim 1 
wherein said second control means and second format 
means for selectively formatting said internal format further 
comprises a means for template matching pattern recogni- 
tion and motion estimation and compensation, comprising: 
means for extracting each of the nxn luminance pixel 

blocks from said previous reference interframe; and 
means for performing template matching operation of said 
nxn reference block within a mxm window of sur- 
rounding pixel blocks residing in said current inter- 
frame wherein m and n are integers and m is greater 
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than n, said template matching generating a motion 

vector for said pixel block in the event of a match; 
means for determining and adjusting said mxm window 

size in the event of no match; 
means for performing pattern recognition on said adjusted 

nxn window sub-image. 

12. A controller apparatus in accordance with claim 1 
wherein said image preprocessing apparatus further com- 
prises a means for identifying an estimating randomly- 
generated signal noise embedded within the input image 
article; and means for subtracting said randomly-generated 
signal noise from said original input image article by means 
of a first sensor and a second sensor, said first sensor 
sensitizing image and noise and said second sensor sensi- 
tizing noise for comparison of said sensors and subtraction 
of said noise signal. 

13. A controller apparatus in accordance with claim 10 
wherein said article means for template matching further 
comprises: 

means for retaining color representation of each signifi- 
cant object of said frame; 

means for storing said color representation; 

means for retrieving said color representation and per- 
forming pel domain transform operations; 

means for converting said color representation from said 
pel domain to said frequency domain in order to 
perform additional frequency domain processing 
means such as discrete cosine transform and fast fourier 
transform. 

14. A controller apparatus in accordance with claim 1 
wherein said band width management means comprises a 
means for generating a network transmission request signal 
for the transmission band width requirement for outbound 
transmission multim^'* articles; 

means for receiving an acknowledgement signal from an 
extension network controller with respect to transmis- 
sion band width requirement; 

means for comparing said acknowledgement signal with 
said outbound transmission signal to determine suffi- 
ciency of transmission band width; 

means for adjusting said outbound transmission signal to 
accommodate said band width availability; 

means for compressing and decompressing said outbound 
transmission signal; 

means for transmitting said outbound transmission signal; 

means for continuous monitoring said outbound transmis- 
sion and said band width availability and selectively 
adjusting said outbound transmission signal to conform 
to said availability of said band width during said 
transmission. 

15. A controller apparatus in accordance with claim 1 
wherein said means for selectively reconfiguring memory 
comprises: 

means for definition of a default internal file format and 
size based upon coder processing and frame memory 
system throughput; 
means far receipt of an external algorithmic coded signal 
during network inbound stage, said means comprising 
a network domain decoder; 
means for receipt of a local origination signal, said means 

comprising a capture processor; 
means for identifying and receiving article types and file 
size during said network inbound stage, said means 
comprising a host processor and its interface to said 
capture processor or said network decoder; 
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means for providing adequate downsampling ratio, 
wherein said receiving article can be conformed and 
reduced to said predefined internal file format and size; 
said means comprising a reconfiguration unit and a 
scaler circuit; 

means for manipulation of said internal file articles, said 
means comprising an article-oriented instruction set, 
and means for storing retrieving, decoding, and execut- 
ing said instruction set by said host processor and said 
pixel processor, 

means for readjustment of said internal file format and 
size in conformance with band width availability, said 
means comprising a band width management unit in 
communication with said scaler circuit, reconfiguration 
unit and said host processor, 

means for adjustment of said internal file format and size 
during interframe coding modes, wherein a plurality of 
motion video foreground articles can be specified to 
proper size, and wherein static background articles can 



10 



is 



a means for receiving a transmit acknowledge signal from 
a network controller and destination receiver indicating 
said band width availability; 

a means for relaying a band width availability signal to a 
band width management unit for comparison of band 
width availability to transmission signal; 

a means for generating an alternate compression ratio for 
said transmission signal compatible with said band 
width availability; 

a means for implementing said alternate compression 
ratio through a host or encoding processor for further 
compression of said multimedia article to meet said 
band width availability; 

a means for transmitting said multime dia article on said 
available band width; and 

a means for monitoring said multimedia article transmis- 
sion in said band width availability during transmis- 
sion. 

18. An apparatus for the continuous management and 
control of transmission band width availability in accor- 



be pretransmitted during intraframe coding modes, said 20 dance with claim 17 wherein said means for generating said 



means comprising a frame differencing circuit and 
motion processor interfaced with said pixel processor, 
and 

means for providing upsampling ratio during 
transmission, wherein said transmitting article can be 
conformed and expanded to a selectively-desired file 
format and size, said means comprising said reconfigu- 
ration unit, and said scaler circuit interfaced with said 
band width management unit 

16. A controller apparatus in accordance with claim 1 
wherein said first control means and second control means 
communicate across the network with external devices 
including pagers, remote control means and host computers 
for the selective exchange of command, data, status and 
control information with said external devices wherein said 
external devices perform selective multimedia applications 
for said controller, said communication means between said 
controlling means and said external devices apprising: 

means for providing command layer protocol control 
functions in the form of predefined commands selec- 
tively interpreted by said control means; 

means for updating and rearranging said predefined com- 
mands; 

means for prioritizing said predefined commands; 
means for providing transport layer protocol control func- 
tions; 

means for data transfer and data tracking in cooperation 
with said command layer protocol; and 

means for providing physical layer protocol control func- 
tions for implementation of initialization termination of 
mechanisms. 

17. An apparatus for the adaptive continuous management 
and control of network traffic condition and band width 
availability for the communication of multimedia articles 
including voice, audio, text, still image, motion video and 
animated graphics between multimedia transmitter and 
receivers wherein said apparatus is compatible with multiple 
analog and digital transmission standards, such as analog 
voice grade line. PSDN, basic rate ISDN, primary rate 
ISDN/T1, LAN, and FDDL the apparatus having a band 
width detection means determining available display band 
widths in accordance with frame rate and run time trans- 
mission band width, said apparatus comprising: 

a means for generating a transmission request signal 
provided with message retaining transmission band 
width for an outbound multimedia article; 
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transmission request signal and said means for generating an 
alternative compression ratio further comprise: 
a means for sensitizing outbound line condition changes 
whereby outbound transmission bit rate is adjusted to 
provide sufficient composition of said multimedia 
article groups, said means for sensitizing said outbound 
line condition comprising a selection of a plurality of 
quality levels for digitally-compressed audio articles; a 
selection of a plurality of quality levels for digitally- 
compressed motion video articles; a selection of a 
plurality of quality levels for digitaUy-compres sed still 
images; a selection of a plurality of frame updating 
rates for digitaUy-compressed motion video articles; a 
selection of plurality of quality levels for digitally- 
coded animated bit mapped or vector graphics articles; 
and a selection of a plurality of options for composites 
of said digital, audio, image, video and animated graph- 
ics articles; and 
a means for selectively monitoring and adjusting said rate 
requirements for said outbound multimedia articles. 

19. An apparatus in accordance with claim 17 for the 
continuous management and control of said band width 
availability, said apparatus further comprising: 

a plurality of predefined program sequences for a plurality 
of transmission line conditions whereby each of said 
program sequences comprises a specific group of mul- 
timedia articles, predefined for said transmission band 
width a vafl ability; and 

a means to store said predefined program sequence and 
reference said predefined program sequence during line 
condition changes. 

20. A method for the adaptive continuous management 
and control of network traffic condition and band width 
availability having a band width detection means determin- 
ing available display band widths in accordance with frame 
rate and run time transmission band width, for the commu- 
nication of ny' l ti m***"* articles including voice, audio, text, 
still image, motion video and animated graphics between 
multimedia transmitter and receivers whereby said appara- 
tus is compatible with multiple analog and digital transmis- 
sion standards such as analog voice grade line, PSDN, basic 
rate ISDN, primary rate BDN/T1, LAN, and FDDL the 
method comprising: 

(a) generating a transmission request signal provided with 
message band width requirement for outbound multi- 
media articles; 
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(b) receiving a transmit acknowledge signal from an can be formatted and sized, said means comprising said 
external network controller and destination receiver reconfiguration unit, and said scaler circuit in interface 
indicating said band width availability; with a band width management unit 

(c) relaying said band width availability to a band width » * sca ^ hle l : ^configurable memory apparatus for 
manag«ment unit fox comparison with band width of * «**■»« ! plm .^ ^P 1 **"*?*^ "* ^l* 1 !^?, 8 
saiZitbound mmtiir^rtdes; *>*£™ ^ to Accordance with claim 21, 

lUrQlCr tTnyii pTi sing " 

<d) generating an alternative compression i ratio for said an integrated parallel processor and storage array whereby 

outbound multimedia articles compatible with said a p^ty of frames of said input image can be 

available band width; 1Q sensitized, digitized and stored; 

(c) relaying said alternative compression ratio to a host or a parallel image preprocessor array for performing real 

encoding processor for further compression of said rimp image encoding operations, said preprocessor 

outbound multimedia articles in conformance with said array including a frame differencing operation whereby 

band width availability; during image encoding operations, said input images 

(f) transmitting said outbound multimedia articles in 15 can be sensitized and compared to develop frame 
conformance with said transmission band width avail- differencing articles whereby said frame differencing 
ability; articles can be further processed for motion estimation, 

(g) monitoring said band width availability and said discrete cosine transform, quantization, and hufEman 
transmission of said outbound mnT tinwrfifl articles dur- variable length coding whereby said motion estimation 
ing said transmission; 20 and motion compensation operations comprise an edge 

(h) generating alternative compression ratios during trans- detection operation for each of said frame differencing 

mission to adapt to changes in said band width avail- ^cles whereby a plurality of detected edge articles are 

abi^ty. derived to identify said motion video objects during 

' , . . . - v . . said motion estimation and compensation operations; 

(1) repeating steps (a) through (h) as required. ^ 

21. A scalable, reconfigurable memory apparatus for 25 . _ . . . _ _ . , fi 

executing a plurality of preprocessing and processing a f f™ extract!^ jeraton whereby extracted specific 

lunctio^for the communication andstorageof imUtimedil ^^f^ <rf said frame differencing articles can 

articles including voice, audio, text, still image, motion „ * , . , , ^ , 

video, and animated graphics between sdectivTmultiinedia ^ A scalaWe reconfigurable memory apparatus for 

transmitters and recovers, said memory apparatus compat- 30 a P 1 ™** of ^oces sin g and post-processing 

ible with multiple standard or custc^coding algcrith- ^ons xn accordance with claim 22 wherein said inpm 

mic signals including H.261, MPEG, JPEG, EDTV or miages comprise moving foreground px^els overlaying still 

HDTV^said apparatus * comprising: back « roimd P ads whcrcbv ba ^f ound ^ 
t o may be characterized singly during lnterframe coding, 
means for definition of a default internal file fonnat and 35 rcquiring ^ c transmission, and said moving foreground 
size bas^pon coder processing and frame memory pixels m dctected usmzlng ^ ^t^f^ differencing 
system throughput; technique in said motion estimation and compensation tech- 
means for receipt of an external network algorithmic nique whereby said parallel processor performs pixel level 
coded signal, said means comprising a network domain operations for an nxn pixel block sub-image, said parallel 
decoder; 40 processor comprising an mxm array of parallel processor 
means for receipt of a locally-originated signal, said elements whereby n and m are integers of equal or unequal 

means comprising a capture processor; value, 

means for identifying and receiving said multimftdia 24. A scalable, reconfigurable memory apparatus for 

article and file size, said means comprising a host executing a plurality of preprocessing and post-processing 

processor selectively interfaced with said capture pro- 45 functioos in accordance with claim 23 wherein said parallel 

cesser and network domain decoder, processor elements can retrieve, decode and execute instruc- 

means for desampling wherein said multimedia article P*?^ 1 its in 

can be conformed and reduced to a predetennined neighboring paraUel proctor elements buffer, said pixel 

internal file format and size, said meaiis comprising a output data can be stored internally and in neighboring 

reconfiguration unit and scaler circuit; 50 r^d ^ocessors in an mterconnection means program- 

7^ . . r ij . * 1 ci c * • « ming each of said parallel processors to a selective group of 

means for manmuktion of said internal file format, said processors whereby pixel data inputs 

means comprising said host processor and a pixel ^control messages can be received from said selected 

processor; parallel processing neighbors and pixel data outputs and 

means for adjustment of said internal file format and size 55 outgoing messages can be transmitted to said selected group 

to conform to band width availability, said means of parallel processor neighbors, 

comprising a band width management unit in commu- 25. A scalable, reconfigurable memory apparatus for 

nication with said scaler circuit, said reconfiguration executing a plurality of preprocessing and post-processing 

unit and said host processor, functions in accordance with claim 24 wherein said parallel 

means for adjustment of said internal file format and size 50 processors can be arrayed and configured as an nxn cellular 
during interframe coding, wherein motion video fore- logic array processor wherein n is a value of between 8 and 
ground articles and still background articles are differ- 32 and said cellular logic array processor can perform 
entiated and separately transmitted, said means com- processing operations for said nxn pixel blocks of sub- 
prising a frame differencing circuit, and motion images including said frame differencing operation between 
processor in interface with said pixel processor; $5 a current frame and a previously-displayed frame whereby a 

means for providing upsampling during transmission group of nxn differencing blocks are identified, registered 

wherein said transmission of said multimedia article and stored; template matching operations for each of said 
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nxn pixel blocks of sub-images, said template matching 
operation extracting ao input image and comparing with a 
previously-stored input image thereby deriving a motion 
vector, pattern matching; motion estimation applied to each 
of said nxn sub-image blocks residing within said 5 
previously-displayed frame; edge detection for each of said 
nxn blocks; feature extraction; Hadamard transform for each 
of said nxn blocks of sub-images whereby each of said nxn 
blocks can be represented by a pixel domain transform 
coefficient and a Hadamard coefficient matrix; and a discrete 10 
cosine transform operation. 

26. A scalable* reconfigurable memory array method for 
executing a plurality of preprocessing and post-processing 
functions for the communication and storage of multimedia 
articles including voice, high quality audio, text, still image, 15 
motion video and animated graphics between multimedia 
transmitters and receivers, whereby said method is compat- 
ible with multiple standard customized coding algorithmic 
signals including H.261, MPEG, JPEG, HDTV or HDTV, 
wherein said method provides a scalable architecture for 20 
co mmuni cation of multime dia data, formatting from internal 
format to external format in compliance with any selective 
one of a plurality of international standardized and custom- 
ized coded algorithms, and band width determination of 
available display band widths according to frame rate and 25 
available transmission band widths, said method comprises: 

defining of a default internal file format and size based 
upon coder processing and frame memory system 
throughput; 

selectively receiving an external network algorithmic 30 

coded signal and local origination signal; 
identifying the multimedia article signal and size; 
downsampling the received multimedia article to a pre- 
defined internal file format and size; 35 
manipulating said internal file format to conform to 

transmission band width availability; 
extinguishing between motion video foreground articles 

and still background articles during interframe coding 

and intraframc coding modes; 40 
upsampling said internal file format to selectively desired 

format and size for transmission in accordance with 

transmission band width availability; 
distinguishing between motion video foreground articles 

and still background articles during interframe coding 45 

and mtraframe coding modes; 
upsampling said internal file format to selectively desired 

format and size for transmission in accordance with 

transmission band width availability. 

27. A scalable, reconfigurable memory array method in 50 
accordance with claim 26 wherein said method for manipu- 
lating said internal file format to conform to band width 
availability and distinguishing between motion video fore- 
ground articles and still background articles during inter- 
frame coding and mtraframe coding modes further com- 35 
prises: 

differentiating between still image background pixels and 
moving foreground pixels; 

performing frame differentiating techniques on an nxn 
pixel block sub-image by means of an mxm array of 
parallel processors whereby n and m are integers of 
equal or unequal value; 

comparing current frame images to previous frame 
images and template matching said images; 65 

deriving a motion vector for each of said frame differ- 
ences; 
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performing pattern matching, motion estimation, edge 
detection, feature extraction and Hadamard transform, 
for each of said nxn blocks of sub-images; 

representing each of said nxn blocks of sub-images by a 
pixel domain transform coefficient, Hadamard coeffi- 
cient matrix and discrete cosine transform. 

28. A method for the application of image preprocessing 
techniques for the interframe coding of multimedia articles 
including voice, audio, text, still image, motion video and 
animated graphics between multimedia transmitters and 
receivers, whereby said method can be integrated with an 
image capturer circuit and band width management circuit, 
said method comprising: 

(a) capturing a local origination still image in accordance 
with frame update requirements established by a host 
processor; 

(b) capturing and storing the complete image frame 
during the mtraframe coding mode; 

(c) comparing and differencing said present captured 
frame to a previous captured frame during the inter- 
frame coding mode; 

(d) storing the frame differences between said present 
frame and said previous frame in blocks and macro 
blocks; 

(e) registering and coding said frame differencing blocks 
or said frame differencing macro blocks into a frame 
differencing bit map; 

(f) converting color space from analog RGB or NTSC 
format into a digital RGB or YUV format; 

(g) retrieving the luminance macro blocks according to 
said frame differencing bit map; 

(h) deriving the appropriate motion vector for each of said 
macro blocks through comparison and selection of the 
minimum distortion among adjacent macro blocks; 

(i) reducing from an external file format to an internally- 
defined format in accordance with a downsampling 
ratio established by said host processor, the reconfigu- 
ration unit and the scaler; 

(j) translating said frame differencing bit map to conform 
with said mternally-defined format; 

(k) performing pixel domain and transform domain cod- 
ing operations for said frame differencing macro blocks 
for said Y, U and V format; 

(1) enlarging from said internal file format to an external 
file format according to an appropriate downsampling 
ratio as established by said host processor, reconfigu- 
ration unit and scaler, 

(m) adding said frame difference to said previous frame in 
order to display and update said current frame; 

(n) repeating steps (a) through (m) as required; 

(o) repeating steps (a) through (n) for live video sequence 
as required. 

29. A method for the application of frame differencing 
techniques, pixel interpolation techniques and annealing 
techniques for the compensation of live motion video 
articles which may include voice, audio, text, still image and 
animated graphics, said motion compensation method pro- 
vides a scalable architecture for communication of multi- 
media data, formatting from internal format to external 
format in compliance with any selective one of a plurality of 
international standardized and customized coded 
algorithms, and band width determination of available dis- 
play band widths according to frame rate and available 
transmission band widths, said method comprising: 
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(a) capturing the scene of a motion video article through 
frame differencing techniques; 

(b) representing the motion video profile of said motion 
video article in an article-oriented instruction set for- 
mat; 3 

(c) defining said motion video profile of said motion video 
article by individual object and relative movement 
association; 

(d) augmenting said motion video profile of said motion 10 
video article by means of rotation, shifting and shuf- 
fling operations; 

(e) reformatting said motion video profile of said motion 
video article to an internal file format; 

(f) enlarging said motion video article by means of 15 
upsampling and reducing said motion video article by 
means of downsampling operations; 

(g) overlaying said motion video profile of said motion 
video article with said voice, audio, text, still image or 
said animated graphics to construct a complete multi- 20 
media presentation; 

(h) defining said motion video profile of said motion 
video article; 

(i) interpolating said motion video profile of said motion ^ 
video article through regeneration of absent pixels to 
further refine said motion video article; 

(j) simulating said motion video profile of said motion 
video article through capturing and recording selective, 
nonconsecutive frame references; 30 
(k) annealing said motion video profile of said motion 
video article through reconstruction of said absent 
pixels, absent motion articles or absent frames, employ- 
ing graphics animation techniques; 
(1) selectively repeating steps (a) through (k) as required. 35 
30. A communication system assembly apparatus execut- 
ing a plurality of control, processing, communication and 
interface functions for the communication and storage of 
multimedia articles including voice, audio, text, still image, 
motion video, and animated graphics between rnultimedia 40 
transmitters and receivers whereby said communication 
system is compatible with multiple standard or customized 
coding algorithmic signals such as H.261, MPEG, JPEG, 
EDTV or HDTV wherein multiple incompatible video cod- 
ing equipment employing different video coding dgorithms 45 
can communicate with each other, said communication 
system providing a scalable architecture for c ommunic a ti on 
of T«iiirimwlifl data, formatting from internal format to 
external format in compliance with any selective one of a 
plurality of international standardized and customized coded 50 
algorithms, and band width determination of available dis- 
play band widths according to frame rate and available 
transmission band widths, said communication system pro- 
viding: 55 
means for network communication processing, said 
means comprising a network domain decoder and net- 
work domain encoder, said means can provide control 
interface to a tdecornmunication network or micro- 
wave network, said means for network cornmunication ^ 
processing including receiving functions for serial 
compressed video bit stream; 
means for transmission processing, said means compris- 
ing a formatter to transform between a C3F/QCIF 
format and selective COIT H.261, MPEG or JPEG ^ 
formats, said means providing conversion functions for 
serial to parallel and parallel to serial, said means 
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further comprising means to encode and decode header 
messages for R261, JPEG and MPEG; 

means for host processing, said means providing a high 
speed interface between said communication assembly 
and an external host, said means further providing a 
system independent communication between said con- 
troller assembly and said external host for the exchange 
sequence of command and data messages, said host 
mfrflps further providing windowing operations permit- 
ting the system memory to be memory mapped into a 
frame memory whereby said host processor can view 
frame memory status and operations in real time; 

means for system host bus, said bus means permitting said 
host processor to control access and communicate with 
network communication processors, transmission 
processors, pixel processors and frame memory; 

means for a communication pipelining subsystem, said 
means providing real time frame formatting, protocol 
controlling transmission and receiving; 

means for pixel processing, said means comprising a bus 
controller and frame memory controller wherein said 
pixel processing means serves as a bus master for a 
video bus wherein said pixel processing means can 
access frame video memory for pixel level operations, 
said means further comprising bit level manipulation 
capabilities for variable length coding and decoding, 
scan formatting conversion and quantization; 

means for display processing, said means for display 
processing performing digital YUV to digital RGB 
conversion, digital RGB to analog RGB conversion, 
digital RGB to VGA conversion and analog RGB to 
NTSC conversion; 

means for capture processing, said means for capture 
processing permitting decoding of analog video input 
formats into CC3R 601, said means for capture pro- 
cessing further decoding said COR 601 into OF 
formulated YUV signals; 

means for frame storage, said means for frame storage 
comprising a twin memory access storage for previous 
frames and current frames according to a formulated 
QF format in order to facilitate frame differencing 
operations; 

means for motion processing, said means for motion 
processing including a means for loading luminance 
signal In frame memory for performing motion estima- 
tion and generating a motion vector for each block of 
storage in frame memory; 

means for transform processing, said means for transform 
processing comparing the differences between the pre- 
vious and the present current blocks or macro blocks 
and coding said differences in discrete cosine transform 
coefficients; 

for video bus, said means for video bus providing 
a high speed, bi-directional, parallel interconnect 
between said frame memory, said capture processor, 
said display processor, said transform processor, said 
pixel processor and said motion processor, and 
means for video subsystem pipelining, said means com- 
prising system-wide direct interconnection between 
said capture processor, said pixel processor, said 
motion processor, said transform processor, said frame 
memory and said display processor. 

* * » # * 
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